The invention relates to a display system having data spectacles, to a corresponding process and to a corresponding computer program.
Nowadays, display devices are known in the form of data spectacles, sometimes also called head-mounted displays (HMDs). These data spectacles are worn similarly to conventional spectacles (eyeglasses) for the correction of poor eyesight and have a display situated in the wearer's field of vision when worn as intended. This display may be designed for being viewed by one eye or for being viewed by both eyes and comprises particularly two partial displays—one for each eye. The display may comprise semi-transparent or non-transparent displays, the wearer of the data spectacles, in the case of semi-transparent displays, being able to perceive the presentation on the display as well as the reality situated behind it. In the case of a non-transparent display, the user sees only the presentation of the display.
Generally, data spectacles can also be used when driving motor vehicles. Particularly data spectacles with semi-transparent displays are suitable for this purpose because, in the case of these data spectacles, the driver can directly see the environment through the display. This makes it possible to see the traffic situation surrounding the vehicle, which is necessary for carrying out the driving task. However, also in the case of semi-transparent displays, there is the risk that the traffic situation is at least transparently covered up by presentations and the view of the traffic situation is thereby impaired. Simultaneously, the presentation may cognitively divert the driver's attention.
The interaction with presentations displayed on the data spectacles can first take place by means of known mechanical input devices or touch-sensitive input devices. For this purpose, for example, a touch pad may be provided laterally at the data spectacles. However, this type of interaction has the disadvantage that one hand or both hands of the user are necessary for this type of interaction. The user's hands will therefore no longer be available for mastering the driving effort, which may be dangerous with respect to driving safety.
Furthermore, the data spectacles can generally also be operated by a command input per voice operation. A process is known from the state of the for this purpose. However, acceptance of the voice input by users is reduced also because of technical difficulties during implementation.
Furthermore, systems have currently become known by which a person's eyes are imaged via a camera and their viewing direction is determined. Such systems are produced, for example, by Tobii Technology Co., Sweden.
In addition, from International Patent Document WO 2013/066634 A1, data spectacles having a camera are known for determining the pupil movement of the wearer of the data spectacles.
It is an object of the invention to control the presentations on data spectacles such that the driver of a vehicle, who is wearing data spectacles, can safely carry out his driving task despite an interaction with the data spectacles.
This and other objects are achieved in one aspect by providing a display system with data spectacles comprising a spectacles frame; a display which is mounted at the spectacles frame such that it is situated in the field of vision of the wearer of the spectacles frame; an image-taking unit, particularly at least one camera configured for providing an image of an eye of the wearer of the data spectacles; a processing unit configured for: receiving the image; determining the viewing direction of the eye in the image; determining the display area in the display in which an activating element is displayed; determining whether the viewing direction of the eye is directed at the display area of the activating element; in response to a meeting of a condition: displaying of assigned contents on the display, which are assigned to the activating element, on a display area that is larger than the display area of the activating element. The condition is that the viewing direction of the eye is aimed at the display area of the activating element, wherein the display area of the activating element is the area of the display which is occupied by the activating element. An activating element may be a symbol, a single-color surface, a miniature preview, etc. The display is mounted at the spectacles frame such that the wearer of the spectacles frame can see presentations on the display. The assigned contents may be a video stream, a photo, a text, a text with images and the like.
By means of an eve movement, particularly a “looking at” the activating element, the wearer of the data spectacles can thereby trigger a presentation of contents on the display, which are assigned to the activating element. The viewing direction is correlated with the image displayed on the data spectacles. Simultaneously, a larger area of the video screen will then be provided to the contents for the display. The user is therefore enabled to control the display of the data spectacles by eye movement. He can therefore retrieve contents according to demand. Since hands are not required for this control, the wearer of the data spectacles and the driver of a vehicle can continue to safely carry out the driving task. This type of interaction is intuitive for the wearer of the data spectacles because he only needs to look at areas on which he wants to have more information.
Instead of working with only one eye, the image-taking unit can also provide an image of both eyes of the wearer of the data spectacles, especially in two partial images. As required, two cameras—one for each eye—are to be provided for this purpose. The viewing direction can then be determined separately for each eye. Only when both eyes have essentially the same viewing direction, will the latter be used as the viewing direction. Otherwise, it can be determined that no viewing direction to be taken into account can be determined and subsequently also no assigned contents are to be displayed.
In a further development, the determining that the viewing direction of the eye is directed at the display area of the activating element includes determining that the viewing direction of the eye is directed longer than a threshold value time at the display area of the activating element. A time period is thereby defined for which the eye has to look into the activating direction before the activation takes place. In this manner, faulty activations are avoided. The threshold value time can also be called a “dwell time”.
In a preferred embodiment, the processing unit is further configured to receive video images from a video camera, particularly of a video camera mounted on a vehicle, which provides images of the surroundings of the vehicle. The presenting of assigned contents on the display includes presenting the video images. In this manner, for example, images of a camera can be shown to the wearer of the data spectacles and driver of a vehicle by cameras at the vehicle. These cameras can take images of surrounding areas of the vehicle which are usually visible by use of exterior and interior mirrors of a vehicle. Looking into the mirrors can therefore be eliminated and replaced by eye movements and presentations in the data spectacles. The driver of the vehicle therefore no longer has to turn his head in order to be able to observe the surrounding areas of the vehicle visible by means of the exterior and interior mirrors. This may be an advantage particularly for drivers whose mobility is restricted.
It may be provided that the processing unit is further configured such that, in response to meeting the condition, an application of the processing unit is activated. The application may trigger the processing of data and the display of data, and particularly a transmission of data to a remote computer by way of a mobile radio data connection. The meeting of the condition therefore results in the activation of an application, thus in a function.
In an advantageous further development, the display is controlled such that several activating elements are displayed in respectively different display areas on the display of the data spectacles, whereby the processing unit is set up for the following: For each activating element: Determining its display area; determining whether and at which display area a look is directed; in response to the determination that the look is directed at the display area: displaying of assigned contents which are assigned to the activating element, at whose display area the look is directed, whereby the assigned contents are displayed on a display area which is greater than the display area of the activating element. A selection is therefore offered to the wearer of the data spectacles, similar to a menu, from which he can choose. Depending on the activating element, at which he is looking, the pertaining content is therefore displayed.
In a typical implementation, it is further provided that the processing unit is configured for the following: Determining that the viewing direction of the eye is not aimed at the display area of the assigned contents; in response to determining that the viewing direction of the eye is not aimed at the display area of the assigned contents, and at the preceding presenting of assigned contents: Terminating the presenting of the assigned contents. It is therefore provided that, as soon as the wearer of the data spectacles averts his look from the display area of the assigned contents, these will no longer be displayed. The assigned contents are therefore no longer superimposed on the look at the surroundings. The assigned contents can therefore easily be shown and hidden again. This is particularly advantageous during the showing and hiding of video streams of cameras directed toward the rearward surroundings of the vehicle.
In a further development, the processing unit is configured for the following: Receiving an instruction for operating particularly a mechanical operating element; determining whether the instruction is essentially received simultaneously or is shifted by not more than a predefined time period for determining that the viewing direction of the eye is aimed at the display area of the activating element; wherein the condition further comprises: The instruction is received essentially simultaneously or shifted by not more than a predefined time period for determining that the viewing direction of the eye is aimed at the display area of the activating element. An activation of the activating element with the display of the assigned content is therefore achieved in that the wearer of the data spectacles directs his look at the display area of the activating element and (essentially) simultaneously actuates, for example, a key. In this manner, it can be ensured that the activation of the activating element is intentional. Furthermore, the safety mechanism of the threshold value time until activation will no longer be necessary. For the activation, the view therefore no longer has to be directed for a minimum time at the assigned display area. Furthermore, this type of input can be intuitively learned by some wearers of the data spectacles. The operating element may be a key at the steering wheel of a vehicle.
The image-taking unit may be part of the data spectacles and be mounted on them. As an alternative, the image-taking unit may also comprise cameras arranged in the interior of a vehicle, which take an image of the head and the eyes of the wearer of the data spectacles.
In a further development, the display system is designed for use in a vehicle and further comprises: A sensing device for determining the head position and head orientation of the wearer of the data spectacles; wherein the processing device is further set up for the following: Determining of free spaces of the display in which the presentation, from the view of the wearer of the data spectacles, is essentially not superimposed on the surroundings outside the vehicle, or the surroundings relevant to the driving task; wherein the display area of the presented assigned contents is selected while taking into account the free spaces. When, for taking images of the eye of the wearer of the data spectacles, a sensing device outside the data spectacles is used, thus, for example, in the interior of the vehicle, the same sensing device may also be the sensing device for determining the head position and head orientation of the wearer of the data spectacles. This takes place by analyzing the image of the head of the wearer of the data spectacles. For this purpose, processes are known from the state of the art.
As an alternative or in addition, the determination of the head position and head orientation can also take place by means of a camera at the data spectacles, which essentially takes images in the direction of the central viewing direction of the wearer of the data spectacles (intended use of the data spectacles). In these images, the structure of the interior (for example, the framing of the windshield or of the side windows) can be recognized and its position can be recognized. A conclusion can be drawn therefrom concerning the head position and head orientation. In this further development, a search takes place for a location for the display of the assigned contents that is as advantageous as possible, i.e. interferes as little as possible. Other sensors, such as position sensors, can also be provided as the sensing device. The assigned contents should be displayed in the location of the display that hinders the wearer and driver the least when he carries out his driving task. It may be provided that the location of the display, thus the display area of the assigned contents, is continuously changed corresponding to the head movement. Typical advantageous display areas for the assigned contents but also the activating elements are in proximity of the road situated in front of the vehicle, above the horizon or over free spaces in the vehicle, for example, on the dashboard.
The processing device may be a computer that can be configured according to a program, thus an electronic computer system, as required, with several separate and spatially separated subunits.
Another aspect of the invention comprises a process for determining a presentation on data spectacles comprising a spectacles frame and a display, characterized by: Receiving an image from an image-taking unit, particularly at least one camera which is set up for providing an image of an eye of the wearer of the data spectacles; determining the viewing direction of the eye in the image; determining the display area on the display in which an activating element is displayed; determining whether the viewing direction of the eye is aimed at the display area of the activating element; in response to the meeting of a condition: Displaying of assigned contents on the display, which are assigned to the activating element, on a display area that is larger than the display area of the activating element, wherein the condition comprises: The viewing direction of the eye is aimed at the display area of the activating element. The process can be further developed corresponding to the above explanations concerning the display system.
In another aspect of the invention, a computer program causes a computer to execute the above-presented or further developed process.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of one or more preferred embodiments when considered in conjunction with the accompanying drawings.